ON A CONJECTURE OF SERRE ON ABELIAN THREEFOLDS 
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En genre 3, le theoreme de Torelli 
s'applique de fagon mains satisfaisante : 
on doit extraire une mysterieuse racine 
carree (J.-P.S., Collected Papers, n° 129) 



Abstract. In this article, we give a reformulation of a result from Howe, 
Leprevost and Poonen on a three dimensional family of abelian threefolds. 
We also link their result to a conjecture of Serre on a precise form of Torelli 
theorem for genus 3 curves. 
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1. Introduction 

1.1. Geometric Torelli's theorem. Let K be an algebraically closed field. If 
X is a (smooth algebraic projective) curve of genus g over K, the Jacobian JacX 
of X is an abelian variety of dimension 5, and JacX has a canonical principal 
polarization A. We obtain in this way a morphism 

Jac : Mg > kg 

from the space Mg of (X- isomorphism classes of) curves of genus g to the space Ag 
of (ii'-isomorphism classes of) g-dimensional principally polarized abelian varieties 
(p.p.a.v.). 

According to Torelli's Theorem, proved one century ago, the map X > (JacX, <d) 
is injective. An algebraic proof was provided by Weil [IH] half a century ago, and 
it is a long time studied question to characterize the image of this map. 
If 5 = 3, these spaces are both of dimension — 3 = g{g + l)/2 = 6. According to 
Hoyt [7] and Oort and Ueno [TS], the image of Mg is exactly the space of indecom- 
posable principally polarized threefolds. Recall that (A, A) is decomposable if there 
is an abelian subvariety B oi A neither equal to nor to A, such that the restriction 
of A to -B is a principal polarization, and indecomposable otherwise. This was a 
problem left unsolved by Weil in [18]. 

Given a principally polarized abelian threefold {A, A) over K , two natural questions 
arise : 

(i) How can we decide if the polarization is indecomposable ? 

(ii) How can we decide if A is the Jacobian of a a hyperelliptic curve ? 

Actually, both questions were answered by Igusa in 1967 [S] when K = <C, making 
use of a particular modular form xis on the Siegel upper half-space (see Th. 13.5.21 
below). 

1.2. Arithmetic TorelH's theorem. Assume now that K is an arbitrary field. 
Then, as Serre noticed in [12 , the above correspondence is no longer true. Let 
{A, A) be a p.p.a.v. of dimension g over K , and assume that (A, A) is isomorphic 
over K to the Jacobian of a curve X of genus g. 

Theorem 1.2.1 (Serre). The following alternative holds : 

(i) If X is hyperelliptic, there exists a model X/K of X and a K -isomorphism 
between the p.p.a.v. (Jac AT, O) and {A, A). 

(ii) If X is non hyperelliptic, there exists a model X/K of X and a quadratic 
character e : Gail{Ks/K) — > {±1} such that (JacA, 0) is isomorphic to 
the twist {A, A)e of {A, A) by e. 

In particular, if e is not trivial, this implies that Jac A is not isomorphic to A over 
K , hut only over a quadratic extension, and [A, A) is not isomorphic over K to the 
Jacobian of a curve. □ 

1.3. Serre's conjecture. Let us come back to the case 5 = 3. Let there be given 
an indecomposable principally polarized abelian threefold (A, A) defined over K . 
In a letter to Top [T7j in 2003, J. -P. Serre asked two questions: 

(i) How to decide, knowing only {A, A), that X is hypereUiptic ? 

(ii) If X is not hyperelliptic, how to find the quadratic character e ? 
He proposed, in the case AT C C, the following conjecture : 

Conjecture 1.3.1. Let {A,X) be an undecomposable principally polarized abelian 
threefold over K isomorphic over K to the Jacobian of a curve X of genus 3. Then 
there is an invariant xi8{A,\) such that 

(i) xi8(^. A) = is and only if X is hyperelliptic; 
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(ii) the character e is the one defined by the action of Gal{K / K) on the square 
root ofxisiA^)- 

We use here the notation xis to emphasize that this conjecture was inspired by 
the resuhs obtained by Igusa, and also much earher by Klein [11] (see the remark 
after Cor. I4.3.3p . In this article Klein relates (up to an undetermined constant) 
the modular form xis ^^nd the square of the discriminant of the quartic X (when 
Xi8(''") 7^ 0). This invariant seemed to Serre a good choice to find this "mysterious 
square root" . 

We plan to answer in the affirmative this conjecture for a family of abelian three- 
folds which are isogenous to the product of three elliptic curves (see Cor. I4.3.2|) . 
This will rely on the work of Howe, Leprevost and Poonen [8^ for which we propose 
a natural rephrasing. For any field K of characteristic different from 2, they con- 
sider abelian threefolds {A, A) defined as a quotient of three elliptic curves (with 
the trivial polarization) by a certain subgroup of 2-torsion points. For this three- 
dimensional family, they make explicit the equation of the related curve and express 
the character e by a invariant T involving the coefficients of the elliptic curves. In 
the first part, we show that T can be naturally interpreted as a determinant. In 
a second phase, we take C C and by uniformization, we express T in terms of 
certain Thetanullwerte of the elliptic curves. Then using the duplication and trans- 
formation formula we express the modular form xi8(A, A) in terms of the same 
Thetanullwerte and compare the two expressions. We also obtain a proof of Klein's 
result in this particular case and give the constant involved (see Cor. I4.3.3p . 
We describe now briefly the different sections. In Sec. [21 we define Ciani quartics, 
go back to the aforementioned results of [8 , and show the relation with Serre's 
conjecture (§ 12. 5p . In Sec. [31 we recall some general facts about abelian varieties 
over C (of arbitrary dimension) and introduce the modular function Xk (§ l3.5p . We 
prove Serre's conjecture in Sec. [H Finally, an appendix gathers some technical 
proofs, in particular the modularity of the form Xk- 

Acknowledgements. We would like to thank J. -P. Serre for interesting discussions 
and S. Meagher for relevant remarks. 

2. CiANi Quartics 

In this section we reformulate a result of [8J on a three-dimensional family of non- 
hyperelliptic genus 3 curves. In particular, this gives a more natural point of view 
on Prop. 15 of [8]. 

2.1. Definition of Ciani quartics. Edgardo Ciani gave in 1899 ^3J a classification 
of nonsingular complex plane quartics curves based on the number of involutions 
in their automorphism group. We describe below the family of quartics admitting 
(at least) two commuting involutions (different from identity). 
Let K be a field with char A' ^ 2, and Sym^{K) the vector space of symmetric 
matrices of size 3 with coefficients in K. Let 



Then 



ai 63 &2 
63 a2 bi 
62 61 03 



Qmix, y, z) = aix'^ + a2y'^ + a^z^ + 2(6iy^z^ + b2x'^z^ + b^x^y"^) 

is a ternary quartic, and the map m h-> Qm is an isomorphism of Sym.^{K) to the 
vector space of ternary quartic forms invariant under the three involutions 

ai{x,y,z) = {-x,y,z), a2ix,y, z) = {x,-y, z), a^{x,y,z) ^ {x,y,-z). 
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The form Qm is the zero locus of a plane quartic curve X„n whose automorphism 
group contains the Vierergruppe V4 — (Z/2Z)^. 

If X,n is a nonsingular curve, we say that X,n is a Ciani quartic and that Qm is 
a Ciani form. Now, E. Ciani [loc. cit.) proved that a plane quartic admitting 
two commuting involutions is geometrically isomorphic to a Ciani quartic (a more 
recent reference is [T]). 

Proposition 2.1.1. If X is a plane quartic curve defined over K, admitting at 
least two commuting involutions, also defined over K , then there is m £ Sym3(_ftr) 
such that X is isomorphic to Xm over K . 

Proof. Let Mi and M2 in PGL3(iir) inducing two commuting involutions of X. 
Then Mf = afi with ai ^ K and det(Mi)^ = af , hence a; is a square and we can 
assume, by dividing Mi by ^/al, that Mf = I. The two matrices Mi commute so 
we can diagonalize them in the same basis : after a change of coordinates, we can 
suppose that Mi are (projectively) equal to 





"-1 





0" 




"1 





0' 


h = 





1 





, h = 





-1 













1 










1 



This implies that a quartic equation Q{x,y, z) — oi X in these new coordinates 
must be invariant by the involutions cti and 172 above, hence, Q is a Ciani form. □ 

2.2. Discriminant of a ternary form. Our definition of a Ciani form includes 
that its zero locus must be a nonsingular curve. This condition is fulfilled if and 
only if the discriminant of the form is not 0. In order to obtain a criterium for this 
condition, we develop an algorithm for the discriminant of a general ternary form. 
The multivariate resultant Res(/i, . . . , /„) of n forms fi, . . . fn in n variables with 
coefficients in a field K is an irreducible polynomial in the coefRcients of /i , . . . /„ 
which vanishes whenever /i,.../n have a common non-zero root. One requires 
that the resultant is irreducible over Z, i.e. it has integral coefficients with greatest 
divisor equal to 1, and moreover 

Res(x^,...,4") = l 

for any (di, . . . , d„) € N". The resultant exists and is unique. There is a remarkable 
determinantal formula for the resultant of 3 ternary forms of the same degree d, 
due to Sylvester; see [6] for a modern exposition and a proof. We give this formula 
in the case d = 3. Then Res(/i, /2, /s) is a form of degree 27 in 30 unknowns. We 
shall express Res(/i, /2, /a) as the determinant of a square matrix of size 15. 
Let Id be the set of sequences v = {ui ,1^2,1^3) with vi + iy2 + i'3 = d. The monomials 
x'^ = x'^^ X2^ x'^^ , where v € Id, form the canonical basis of the space Vd of ternary 
forms of degree d, which is of dimension {d + l)(d + 2)/2. 

For any monomial x'^ G V2 with i/i+i^2 + '^3 = 2, we choose arbitrary representations 

f^ = a^r + '/M + 4' + V.2 + 4V<,3 (1 < * < 3), 

for 1 < i,j < 3. Such a representation is 



where fij are forms of degree 2 
always possible, although not unique. Now we define 



Six") = det 



fi,i fia /i,3 

/2,1 f2,2 /2,3 

/3,i /a, 2 /a, 3 



Note that this determinant is indeed a ternary form of degree 
(2 - ui) + (2 - 1/2) + (2 - 1^3) = 6 - 2 = 4. 
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Since the monomials x'^ with v Cz I2 make up a basis of V2, we have thus defined a 
hnear map S : V2 — > V4. Wc consider the linear map 

T -.Vi xVi xVi XV2 y Vi 

given by 

Tih, h, h,g) = hh + hf2 + hh + S{g). 
Now one proves that the determinant of T is independent of the choices made in 
the definition of S, and Sylvester's formula holds : 

Res(/i,/2,/3) =detr. 

Generally speaking, the matrix of T involves 864 monomials. Now, let Q be a 
ternary form of degree d, and X be the plane projective curve which is the zero 
locus of Q. Call qi,q2,q3 the partial derivatives of Q. The discriminant of Q is 
DiscQ ~ Res((7i, ^2, 93)- It is a form of degree 3{d — 1)^ in the coefficients of Q, 
and X is non singular if and only if DiscQ 7^ 0. The discriminant is an invariant 
of ternary forms : if 5 G GL3(A'), then 

(1) Disc(Qo5) = (det g)"" Disc Q, where w:^d{d- 1)^ 

If Q is a quartic form Disc Q is a form of degree 27 in the coefiicients, with w = 36 
in llj. Applying Sylvester's formula, we get : 

Proposition 2.2.1. Let m E Sym^{K) and Ci = ajUk — hi is the cofactor of Ui 
for 1 < i < 3. If {qi,q2,Q3) are the partial derivatives of the ternary quartic Qm, 
then DiscQm — 2^^D(m), where 

D(m) — ai 02 03 (ci C2 ca)^ det(m)''. □ 

Note that this result was obtained by Edge [1] , in a more intricate way. 
We denote by S the set of ni e Sym.^{K) such that 

010203 ^ 0, C1C2C3 ^ 0. 

Now, Prop. 12.2.11 implies that the curve Xm is nonsingular if and only if m belongs 
to the set = Sn Gial^)- 

Lemma 2.2.2. The map m !—)■ Qm from to the set Q of C'iani forms is a 
bijection. □ 

The automorphisms of X,„ induce a simple description of its Jacobian. In order to 
make it explicit, we need to introduce a certain product of elliptic curves. 

2.3. Product of elliptic curves. We introduce the following notations : let 

E,:y^ =xix^ -4h,x-4c,), (h, e K, c, e K"") (i = 1,2,3), 

three elliptic curves with (0, 0) as a rational 2-torsion point. The discriminant of 
Ei is Ai = 2^^c|(5i, where 6i — + Ci E . We assume that there exists a square 
root p S of 6{A) = S1S2S3, that is, A1A2A3 is a square in K. We denote by A 
the set of products Ei x E2 x E3 of such curves and we define 

A={l=(A,p)eAxi^^ I p2^5(A)}. 
If j4 e A, we put Oi — p/di and 

[oi 63 b2 
Mat(A) =^ 63 02 bi € S. 
h bi a3_ 

Conversely, a matrix m e S defines an abelian threefold A{m) £ A, which is the 
product of the curves 

E, : = x{x^ - Ab^x - Ac^) (^ = l,2,3). 
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Then 

St ^ bf + Ci = ajGk € , Ai A2 A3 = (2^* 010203010203)^, (5(A) = (010203)^. 
We define p{m) — 010203, and Ab(m) = {A{m),p{m)) E A. 
Lemma 2.3.1. The maps 

Mat: A > S, Ab: S » A, 

ore mutually inverse bijections. □ 

The two lemmas 12.2.21 and 12.3.11 provide a natural map from the set Q of Ciani 
quartic forms to A. This map has actually a geometric meaning, and in order to 
explain it, we introduce the following notation. If m e Gh^lK), we denote by 
Cof(m) the cof actor matrix of m, satisfying 

m.*Cof TO = det m.I, det Cof to = (det m)^, Cof Cof to = (det to).to. 

Let Qm be a Ciani form associated to to g and X^. be the corresponding Ciani 
quartic 

: Qyn(x,y,z) = Fm{x^,y^,z^) = 0. 
By quotient, we get three genus one curves 



Ci := X„/ < 1,(71 > 
C2 := X^/ < 1,(72 > 

C3 := Xm/ < 1,(73 > 



F(yz,a;^J/^) = 0, 
F{zx,y^z^)^0, 
F{xy,z^,x^)^Q, 



where ct^ (i = 1, 2, 3) are the involutions of Xm- Another change of variables maps 
the genus 1 quartics Ci to the elliptic curves 

Fi:y'^= x{x^ - 4djX - 4oi det(TO)), {i = 1, 2, 3). 

In this way, we get a map 

if> : Xrn » Brn = Fi X F2 X F3. 

Let us now look more closely at Bm- The identity 

Cof Cof TO = (det to). TO 
implies that the cofactor of Ci is o^ det to. Hence, 

Ab(Cof to) = {B,n, Cl C2 C3). 

Since the Jacobian is the Albanese variety of Xm, we get a factorization 

J aC XjYi Sjn 

where t is a canonical embedding. Since the images of the regular differential forms 
on Fi make a basis of those on Xm, we obtain: 

Proposition 2.3.2. The map 

$ : JacAm > A(CofTO) 

is a (2, 2, 2)-isogeny defined over K . □ 
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The correspondences 

m > Cof 1 



Qm > A{Cofm) > JacXm 

lead to a commutative diagram, where Q is the space of Ciani quartics over K: 





Cof 










Q 


" Jac " 
> 


A 



In the next section we describe the kernel of the isogeny 

2.4. The theory of Howe, Leprevost and Poonen revisited. The previous 
isogeny can be made more precise as we recall from [8 . There are some differ- 
ences between their notation and ours, see the remark at the end of Sec 12. 51 for a 
comparison. Let us introduce couples {A,W), where: 

(i) AeA as defined in § O 

The Weil pairings on the factors combine to give a non degenerate alternating 
pairing 62 on the finite group scheme A[2\ over K. 

(ii) is a totally isotropic indecomposable subspace of A[2] defined over K. 

Choose a basis {Pi,Qi) G £'^[2], that is, a level 2 structure on Ei. This defines 
a level 2 structure on A. In |Si Lem.13] it is proved that after a labeling of the 
2-torsion points we can write 

^2^ w^l (0,0, 0), (0,Q2,Q3), (gi,0,Q3), (gi,Q2,0), 

^' \{Pl,P2,Pz), (Pl,i?2,i?3), (i?l,P2,i?3), (i?l,i?2,P3) 

with 

Qi = (0, 0), P,; = (0, 26,; + Ri = {Q,2bi- Pi), piP2P3 = Pw, 

and the four possible choices of pi, p2, Ps leading to the same value of P1P2P3 give 
the same subgroup W. Conversely, if {A, p) S A is given, if we choose pi,p2,p3 
in such a way that P1P2P3 = p, and if we define Pi and Ri as above, then we can 
define a subgroup Wp by 

Lemma 2.4.1. The map {A, p) 1-^ {A,Wp) from A to the set of couples {A,W) as 
defined above is a bijection. □ 

We take on A ~ A{m) the principal polarization A which is the product of the 
canonical polarizations on each factor. Then we have a commutative diagram 

A 



8 d°8 



A' (.4')^ 

with a unique principal polarization A' on A' = A'(to) = A{m) /Wpi^yny From 
[SI Prop. 15] we get : 

Theorem 2.4.2. The composition of the isogeny $ and of the projection tt leads 
to an isomorphism of p. p. a. v.: 

JacX™ > A'(Cofm). □ 
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As a corollary, for any m, the p. p. a. v. {A',X') is indecomposable. 

The reverse direction is more interesting and will give an algebraic answer to Serre's 

conjecture. 

2.5. Relation with Serre's conjecture. We need the following elementary lemma 
from linear algebra. 

Lemma 2.5.1. The map m i— > Cof m induces an exact sequence 

1 y {±1} y GLsiK) G'^'^{K) > 1 

with 

G''2(X) = {meGL3(if) I detmeX'^^i^ □ 

Let {A,p)^A^k and m = Mat (A). Denote 

T(I) := det(m) = 2616263 - Pij- + T - 

oi 02 03 

Theorem 2.5.2. The following results hold. 

(i) if '[{A) = 0, that is, me S \ S^, there is a hyperelliptic curve X of genus 
3 such that A'(m) is isomorphic to the Jacobian of X . 

(ii) ifT{A) 7^ 0, that is, rn G S^, then there exists a non hyperelliptic curve 
of genus 3 defined over K whose Jacobian is isomorphic to A'(m) if and 
only ifT{A) is a square in K. 

Proof. The first part is [H Prop. 14] where the hyperelliptic curve is constructed 
explicitly. For the second part, if det(m) is a square, then using Lem. 12.5.11 we see 
that there exists a matrix m! e such that m = Cof(m') and we apply Th. 12.4.21 
If d = det(m) is not a square, let = dm and A^ = Ab(mrf). A{md) is defined 
by 

E, : ?/2 ^ x{x^ - -ibidx - ic,d^) {i = 1, 2, 3), 
Thus A(md) is a quadratic twist of A{m). Now, det(mrf) is a square, so there exists 
to' such that Jac{Xm') is isomorphic to A'^md). Since A'{md) is a quadratic twist 
of A'(m) and is the Jacobian of a non hyperelliptic curve, Th. 11.2.11 shows that 
A'{m) cannot be a Jacobian. □ 

Corollary 2.5.3. With the same notation as above: 

(i) ifT{A) G K^'^, there is an isogeny defined over K 

JacAr„j' > A(to), Cof to' = to, 

(ii) //T(^) ^ K^"^ , there is an isogeny defined over K 

JacXm' > A{md) Coim'^dm. □ 

We hope to give in a near future a geometric interpretation of the connection of 
Serre's problem with the determinant of certain quadratic forms in the general case. 

Remark. In [8], Howe, Leprevost and Poonen write the elliptic curves 
y"^ = x{x'^ + AiX + Bi) aYecAi,Bi€K {i = 1,2,3). 

So 

A, = -46„ Bi = -4ci, 

d, = -(A, + 2x{P,)) = 46, - 4(6, + p,) = -Ap„ df = A,. 
And the factor 

ToiA) = did2d3(^ + ^ + ^-^)- 2^1^2A3, 

Ai l\2 ^3 
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which is 



T^{A) = 64[-p(^ + M + M _ 1) + 2616263 
Ol 02 03 



We then have To{A) = 64T(yl). 



3. Complex abelian varieties 

We recall in this section some well known propositions on abelian varieties over C 
and fix the notation. 



3.1. The symplectic group. If y is a module of rank 2g over a commutative ring 
R and if _B is a non-degenerate alternating bilinear form on V ^ a basis {ai)i<i<2g 
of V is said symplectic if the matrix {E(ai, aj)) — J, where 



The group of matrices 



J 



M = 




1„ 



A B 
C D 



e SL2giR) 



such that M.J}M = J is the symplectic group Sp2g(i?). It acts simply transitively 
on the set of symplectic bases of V . 

Lemma 3.1.1 (O Lem.8.2.1]). If M G Sp2g(i?) the following conditions are equiv- 
alent. 



(i) MeSp2,(i?). 

(ii) *A.C and *B.D are symmetric, and ^A.D — *C.B — Ig 

(iii) A^B and C^D are symmetric and A}D — BKJ ~ Ig. 



□ 



The group Sp2g(i?) is the group of i?-rational points of a Chevalley group scheme 
Sp2g, which contains certain remarkable subgroups defined as follows. The reductive 
subgroup M of Sp2g is the subgroup which respects the canonical decomposition 
I?3 0)13. Elements of M(Z) are 



M 



A 

*v4-i 



A e GLg(Z). 



The unipotent subgroup U is the stability group leaving pointwise fixed the canon- 
ical totally isotropic subspace Vo, which is the first direct summand in the standard 
decomposition. Elements of U(Z) are 



U 







B 

In 



^B = B, Be Mg(Z). 



The unipotent subgroup V opposite to U is the stability group of the second direct 
summand in the standard decomposition. One has V = *U = J.XJ.J~^. Elements 
of V(Z) are 



V = 



1. 

c 







*C = C, CeMg(Z). 



The subgroup P = M k U is the parabolic subgroup of Sp2g normalizing Vq , and 
P is actually a maximal parabolic subgroup. Elements of P(Z) are 



A B 




A.'^B = B.% A e GLg(Z), B e Mg(Z). 
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3.2. Abelian varieties. Let = [wi . . . W2g] G Mg^2ff(C), where Wi, . . . , W2g are 
columns vectors giving a basis of on M. It generates a lattice 

A = ^ £g_ 

Let 5ig be the set of matrices € ]V[g.2g(C) satisfying the Riemann conditions 

n.j.*n = o, 2i{Ti.,r^ > 

(> means positive definite). We call such a matrix f2 a period matrix. If f2 S 
3ig, the torus = C^/A is an abelian variety of dimension g with a principal 
polarization A represented by the hermitian form H = 2i{n.J^^.*^l)~^ (see [21 
Lem.4.2.3]). 

The group GLg(C) acts on the left on Jig. If we write 

= [{wi . . . Wg) {wg+i . . . W2g)] = [^1 ^2], where D.i e Mg(C), 

we get W.[Qi ^,2] = [W.Qi W.Q2] for any W £ GLg(C). This action induces an 
isomorphism of p. p. a. v. In particular if we choose W — ri^^, we see that is 
isomorphic to the p. p. a. v. 

and n £ 5ig a and only if T{fl) belongs to the Siegel upper half plane 

Hg = {t e Mg(C) I V = r, Imr > 0} . 

We call a matrix r G Mg a Riemann matrix. The Siegel modular group Tg — Sp2g (^) 
acts on the right on 3?„: if f2 e 3?„ and if M G r„, 



n.M = [ni ^2] 



A B 
C D 



= [iliA + f^aC niB + n2D]. 



This action corresponds to a change of symplectic basis. The group Tg also acts on 
the left on the Siegel upper half plane : if r e Hg, we denote 

M.T = {At + B){Ct + D)-\ 

Both actions are linked by 

(3) M.T{n) = t{VL.%I). 

3.3. Isotropy and quotients. For any maximal isotropic subgroup V C Fj^, we 
have the transporter 

Trans(y) = {M e Sp2g(F2) | AfVo = V] , 

Vo being the canonical maximal isotropic subgroup generated by the vectors ei , . . . , Cg 
of the canonical basis. Since Sp2g(F2) permutes transitively the maximal isotropic 
subgroups of F2^, the transporter is a left coset: Trans(V^) = AfoP(F2), for any 
Mo e Trans(y). Hence, the set of maximal isotropic subgroups is the quotient set 
Sp2g(F2)/P(F2), a set with 135 elements if g = 3. 

Let now Q. S Dig, A = 02?^ and {A, A) — (^si, H) be the corresponding p.p. a. v. of 
dimension g. The linear map a : 1?^ such that 

a{x) ~ 2^--'^ 

defines a level 2 symplectic structure on A[2\, that is, an isomorphism 



a 
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and if y C is a maximal isotropic subgroup, the same property holds for 
W = a{V) C A[2]. If TT : ^ C^/A is the canonical projection, the lattice 
Aw — 7T^^{W) is associated to A/W as the following diagram shows: 

> Aw/A > CVA > C^/Aw > 



> W > A > A/W 

We define 

Trans(VF) = {M e Tg \ Af(mod2) e Trans(t/)} . 
We introduce now the congruence subgroup 

\A Bl 



C D 



e Tg I C = 0(mod2) 



This is the transposed subgroup of the group rg(2), see § lA.ll From Prop. lA.l.ll 
we deduce that ro,g(2) = P(Z).rg(2), hence 

Trans(VK) = Mro,g(2) 

for any M e Trans(M^). 

Proposition 3.3.1. With the previous notation, if t = T{n) then ^^I.t is a 
Riemann matrix of the p.p. a. v. A/W for all M € Trans(VF). 

Proof. If AI e Trans(W), modA is generated by the vectors 

2 2 ^ 

and the matrix 

n' = n.M.H, H= 2^ 

generates Aw- Using Q, we get 



lg_ 



T{n') = \M.H).T = ^*Mt. 

By [TH Prop. 16.8], the polarization 2A of A reduces to a principal polarization A' 
on A' ~ A/W. This last corresponds canonically to il' since 

2i{TF.j-\*n')-^ = 2i(nMHj-^VM*n)-^ = 2 • 2i{nj-^*n)-\ 

□ 

3.4. Theta functions. We recall the definition of theta functions with (entire) 
characteristics [e] = [H] where £1,62 G Z", following P]. The (classical) theta 
function is 

The Thetanullwerte are the values at z = of these functions, and we denote 

{r)^^m (0,r). 

We now state two formulas. 

Proposition 3.4.1 (duplication formula, see ^ Cor.IIA2.1] and [TUl IV.th.2]). 

Let \ £2] and he two characteristics and t € Hg. Then 

(4) dm{r/2)dm{T/2)^ J2 (-ir'-^M w-^U-^] 

M6 
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A B 
C D 



e Sp2<,(Z). We 



Dei-Ce2 + {C*D)Q 
-Bei+Ae2 + (^'B)o 



The second is called transformation formula. Let M — 
let M acts on the characteristics in the following way 

[M.e] = M. m 
where Pq denotes the diagonal of the matrix P. 
Proposition 3.4.2 (10, V.§.2]). 

^[M.e]{M.T) = k{M) ■ w'^[-i.«2iW .j{M,Tfl^ ■ d[e\{T) 
where K{M)'^is a root of 1 depending only on M, uj — e"!"^, 

jiM,T) = det (Cr + Z?) 

and 

(M) = si'DBsi - 2ei'BCs2 + £2*CAe2 - 2{Dei - Ces) • {A'B)c 



□ 



We will need a slightly modified version of the previous result. 

Corollary 3.4.3. For any characteristic and for any M € Sp2g(Z) we have 

(5) iM.r) = ciM,r)..-^^.^^^^^^-^.,^:i^ (r) 

where 



c{M,t) = k{M-^)-^ •i(M,T). 

Proof. To inverse the action on the characteristics, we let t' = M~^.t in the trans- 
formation formula. Note that 



(C*D)o' 
(A*B)o. 



Thus we get the action on the character- 



and that [M.e] = [*Af-i(^i)] + 
istics. For the factor j{M,T) note that 

j{M, t) ^ dct(CT + D)= dct{CM-\T' + D) 

= det(C(*£'r'-*B)(-*Cr' + *A)-i+D) 

= det(C(*i:'r' - + Di-'^Cr' + det(-*Cr' + 'A)-^ 

= deti-'Cr' + 'A)-^ ^j{M-\t')-^ 

using Lem. 13.1.11 □ 

Corollary 3.4.4. Let fl ~ [fli ^1,2] be a period matrix and t — t{VL) — ilj^^fii g 
Hg. Let n' = = [n{ r^y. Then 

j{M,T{n)) = det(r22)~^ ■ det(r2'2). 

Proof. We compute 

det(CT + i:') = det(r*C + *L») 



det(f72)"^ •det(f72), 



the last expression coming from ((3|). 



□ 
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3.5. The modular function Xk- Recall that a characteristic [e^] is even (resp. 
odd) if ei.e2 = (mod 2) (resp. £1.^2 = 1 (mod 2)). Let Sg (resp. Ug) be the set 
of even (resp. odd) characteristics [Jj] with coefficients in {0, 1}. It is well known 
that 

#^5 = 29-1(2^ + 1), #[/,, = 2^-1(29-1). 

Let 51 — [ill ^12] G and r = fl2^^ili e be a Riemann matrix. For g > 2, we 
denote k = ^Sg/2 and we are interested in the following expressions : 

Xfc(r) = n mir). 
seSg 

Recall that a function / is a modular form of weight w for the congruence subgroup 
r e Tg if for all t e Hg and M e F one has 

fiM.r)^jiM,rrfir). 

Using Cor 13331 we get 

Corollary 3.5.1. Let f be a modular form of weight k for F on Hg. For Jl = 

[fli fl2] € iKg, we define t — Q2^^i G Riemann matrix and 

f{n) dct(r!2)-^ • /(r). 

Then for all M € F 

/(fl.M) = /(fl). □ 

In his beautiful paper Igusa proves the following result [loc. cit., Lem. 10 & 
11]. Denote by E140 the thirty- fifth elementary symmetric function of the eighth 
power of the even Thetanullwerte. 

Theorem 3.5.2. For g > S, the product Xfe(T) is a modular form of weight k for 
the group Fg. Moreover, If g ~ i and t G Hs, then: 

(i) At- is decomposable if Xisi^) — 5]i4o(r) — 0. 

(ii) At is a hyperelliptic Jacobian if Xi&i^) = ^.''T-d Si4o(r) 7^ 0. 

(iii) At is a non hyperelliptic Jacobian if Xis{t) 7^ 0. □ 

This theorem gives an answer to the two questions raised in Sec ll.ll over C. 

In the sequel, we will need the following result to prove the independence of our 
results from the choices we will make. The proof is the case g = 3 of Th. IA.1.21 

Proposition 3.5.3. The product t i~f Xisi'^''') is a modular form on H3 of weight 
18 /or F0(2). 

4. Comparison of analytic and algebraic discriminants 

In this part, we make the link between the algebraic result Th l2.5.2l and Serre's 
conjecture on the modular function Xis- To do so, we first compute a quantity 
related easily to T{A) in terms of the Thetanullwerte on the elliptic curves. Then, 
after a good choice of a symplectic matrix N (related to the subgroup W we use 
for the quotient), we compute xi8((*A^.t)/2) in terms of the same Thetanullwerte. 
Thus, we express xig on the quotient A/W. Finally we compare the expressions to 
get Serre's conjecture. 
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4.1. Expression of the algebraic discriminant. We come back to the hypothe- 
ses of § 12.31 and spcciahzc to the case K C C. Let A — Ei x E2 y. ^ k, where 

Er-V^ ^x{x^ -Ahx-Aci), {heK,c,eK'') {i = 1,2,3). 

We choose a root p of d{A), and put m — Mat(yl) with A ~ {A, p). Let 

X(r7i) :— (fli 02 03)* (ci C2 03)^ detm. 

Since T(A) = detm, T(^) is a square in K if and only if X(m) is a square in K. 
The function X appears naturally in our problem since it is related to the function 
D(m) (Prop. by 

(6) X(Cofm) = D(m)2, 

and this reflects Serre's conjecture according to Th l2.4.2l In order to determine 
the expression of X{m) is terms of the ThetanuUwerte, we use the following uni- 
formization. The curves Ei can be written as 

E{uou.oJ2i) ■■ = x{x + 4^ g] infnx + 4^ [?] in)^), 



with [ujii ijj2i\ E 3?i and = We identify Jlf with the set of matrices 
such that 








( t^21 








^12 















^13/ 


^0 





1^23 



T = T{n) = n^^ni = 



Ti 0' 
r2 
r3 



We define 



A{n) E{ujii,uj2i) X £^(^12,^22) X E{uji3,uj23), 



111 
000 



ir). 



64(detr22)2 

This defines an element A (il) {A{Vt) , p{Vt)) e A, and a matrix m(ri) := Mat(A(r2)). 
For 1 < i < 3, denote 

%-79[0](t,), ^u^^[h]{n), ^2^=^[1]{T^). 

The coefficients of A{n) and m{il) are 



4 iL02jL02kV n^ 



4 4 %^'2., 



where {i,j,k) is a cyclic permutation. The determinant of m(f2) is expressed as 
follows. Let 



a — 't?oi''^02^23i 



,<)2 o2 n2 
(^01 ''22 '^03' 



,92 ,92 ,92 
t'21'^02 '^03! 



,92 ,92 ,92 
"21 "22 "23! 



Then 



Ri = (a + b + c + d)(a + b- c-d)(a-b-c + d)(a- 
detm{n) = — — — — — ^7— • Ri. 



2^-n-=iK-«-^y) 
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Thus we get 

X{m{n)) = ( 



2^-nti^ 



2i i=l 



2«-nti' 



'2i 1=1 



2^-ntiK-«-^i.)) 

54 3 
240 



Ri) 



_54 •J 

• det{n,r'' ■ {Y[K.^UK. - ^t^f) ■ Ri- 



i=l 

4.2. The subgroup VF. With the notation of Sec. 12. 4|, we can always assume the 
following correspondences 

Pt^—, Qi'^—, Ri^Pi + Qi^ ^ 

for the points of Et . The characteristics associated to the points of W (see § 12. 4p 
are 

It defines a maximal isotropic subgroup y of F2 . A basis of V over F2 is given by 
the three vectors 

The matrix 



roooi roooi roooi roooi riiii riiii riiii riiii 
loooJ ' loiiJ ' lioiJ ' liioJ ' loooJ ' LoiiJ ' lioiJ ! liioJ 



N 



10-10 

1 

1-10 

1 
-1 -1 1 

1 

belongs to Fa and satisfies N.ei = ai (mod 2) if 1 < z < 3, thus N e Trans(W^). 
The set 



rg(i,2) 



A B 
C D 



e Tg I iA.'B)o = iC.'D)o = 0(mod2) 



is a subgroup of Tg, and is a character of rg(l, 2) [10, p. 181]. 

Lemma 4.2.1. The matrices N and *7V are in Tsil, 2), and k{N)^ = K{*'N f = ±1. 

Proof. We have N — LQ, where 



L 



A 




with A 



-1 1 
1 
-1 1 



and 



1 

1 

1 

-1 

-1 

1 



One checks easily that L, %, Q, *Q belong to r3(l, 2), hence, N and *iV are in r3(l, 2) 
as well. If 

\A B 
D 



M 



€ P(Z), 
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then K.{Mf = detD, see [H Lem. 7, p. 181]. 

with S 



Now 



S 
S 



-1 



From this we deduce that 

K{Lf = det^ = : 
hence, k{NY = k{*N)'^ = ±1. 
Proposition 4.2.2. Let n' = nNH. Then 



2\2 



dots' = 1, 



□ 



1. 



is a Riemann matrix for A' {m) . Moreover, the value xis{^') is independent on the 
choice of N £ Trans(VF). 

Proof. The first assertion conies from Prop. 13.3.11 the second from Prop. 13.5.31 □ 

4.3. Expression of xisi^') as a discriminant. Our main result in this section 
is the following 

Theorem 4.3.1. Let 17 G 3?^ and A{fl) be the corresponding abelian threefold, let 
m = m(Q) € S be the associated matrix, and D,' Cz ^3 be a Riemann matrix of 
A(n)/W. Then 



(f)''•Xl8(^^') = X(m). 



and 



Proof. The strategy is the following. Let TV be the matrix defined in 
define r' = ^N.t = 2T{n'). 

(i) Pair the ThetanuUwerte in t'/2 such that one can apply the duplication 
formula (jl]). We then obtain expressions in terms of ThetanuUwerte in r'. 
Such a pairing is not unique and one makes here a choice which allows an 
easy comparison of the final formulas. 

(ii) For each of the ThetanuUwerte in t', apply the transformation formula ^ 
to obtain an expression in r. 

(iii) Finally, since r = diag(ri, T2, ra), we get 

HZlllH {r)^f[nii] in). 



Let 



±det(02)"Met(r22; 



c{N) = kCN-'^)-^ det(f72)"^ detin'^) 
by Lem. [iXTl 

Applying steps Q to (jml with the software MAGMA (see http : / / iml . univ-mrs . f r/'^ritzenth/programme/chec 

we get the following 18 identities, where we write 

^ d ^ [p] = ^ ra (^72)1? (^72), c = c{N). 

We make the pairing in such a way that the expressions of ■& [^^2] ^ I 
contain terms. The first four are, with the preceding notation, 

c(a + b + c + d) 

c(a + b - c - d) 

— c (a — b — c + d) 

— c (a — b + c — d) 



roooj 



do not 



,9 roooi ,9 [0001 
LoooJ " looiJ 

,9 [0001 ,9 [0001 

loioJ ^ loiiJ 

,9 [0001 ,9 [0001 

aooJ aoiJ 



,9 [0001 ,9 [0001 

and the remaining 14 are 

^ [808] ^ [80?] = 2c (l?01^?2ll?02l?22^§3 + l?0ll92H902l?22l?^3) 
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^ IhD ^ [h°0l] = 2c (l?^i^02l?22l?03^?23 + ^?L^?02^?22^?03^?23) 

^Cl^Ii?)?] = 2c(^92,l?21^?^2^03^23+l901^21^?L^^03t?23) 

^ d = 2C (z?^i^2ll?02l?22C - l9oi^21^?02^?22^?^3) 

^ [h%] ^ [J??] = 2C (l?2^^902^?22^903^?23 " ^?L^?02^?22t?03t?23) 

[}}8] ^ [}}?] = -2C (^01^21^^2'903'923 - ^011^21^^2^03^923) 

^[^°oh]^\?>%] = 2c(l?oH?ll^?02^9l2^?03^?13) 

^[%o]^[fio] = 2c(l?oH?lH?02^?12^03^?13) 

'J [lio] [lOo] = 2c(t?ii^921^?12^?22^?03^13) 

'^d'^Cl = 2c(t?01^^11^?12^?22^13^23) 

[OOO] ^ [lio] = 2C (Z9n^2ll902l9l2l9l3l?23) 

[oil] ^ [ill] = -2c(^11^21^12l?22l?03l?13) 

'J [on] ^ [lOl] = -2c(l?11^21^02l?12l?13^23) 

'J [l§l] ^ [l?l] = -2c(l9oi^ll^l2l922l9l3^23) 

Denote by R[ the product of the first four lines. Obviously R'j^ — c{N)'^Ri. Calling 
R'2 the product of the last fourteen lines, we get 



So 



o40 

Xi8(r72) = R'lR^ = . ^(^)i8 . / det(f]2)^' ) • X(m) 



2X54 



det(f}^)i« •X(m), 



which is the expected result. □ 

Since X(m) is equal to T{A) up to a square, Th l2.5.2l and Th l4.3.ll show Serre's 
conjecture. 

Corollary 4.3.2. Let K C C and m E with coefficients in K. Let A'{m) he 
the associated abelian threefold and Q,' he one of its period matrix. Then 



.2, 

if and only if A'(m) is the Jacobian of a non hyperelliptic genus 3 curve. □ 

In other words, Serre's conjecture is true for our three dimensional family A of 
abelian thrcefolds. 

Corollary 4.3.3. // m G 5^ and ilm is a period matrix associated to the non 
hyperelliptic genus 3 curve Xm with Ciani form Qm then 

Xl8(f^m)=[^j •Disc(g™)2. □ 

Proof. Using Th l^X^ and ^ we get 

(|)'^Xl8(^^m) = X(Cofm) 

= D(m)2 = (2-54 .Disc Q„)2. 

□ 
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Remark. When m g S \ S^, the abehan variety A'(m) comes from a hypereUiptic 
curve and the above formula degenerates. However in |13j and [S] we find a beautiful 
formula for the hypereUiptic case in every genus. Let 

C:Y2= a2g+2X^s+^ + ... + «□ = a2g+2(^ - ai) ■ ■ ■ {X - 023+2) 

and 

3<k 

They define also a modular form on Mg 

EST 

where T is a certain subset of even theta characteristic. One has 
where 

and T = T{n) for a certain period matrix fl = [fli, ^2] of Jac(C). 

Remark. Denote by the 15-dimensional affine open set of ternary quartics. Felix 
Klein proved in 1889 that there is a map 

such that if fl{Q) = [fli il2] and X : Q = 0, then JacX = Aqjq) and 

Xl8(^^) = cDisc(Q)2, 

with some unspecified constant c G C. We prove here that c = (l/27r)^''. Using this 
precise version of Klein's formula, it is almost obvious to extend our theorem to the 
general case. However, we did not include it, for we think that a good presentation 
should include a modern proof of Klein's result. We plan to do this in a forthcoming 
article. 



Appendix A. 

A.l. Modularity of Xk- Let 

rg(2) = {M G Tg I M = l2g(mod2)} 
an recall that the sequence 

i^r,(2)^r,^Sp2g(F2)^i 

is exact. We introduce the congruence subgroup 

e Spg(Z) I B = 0(mod2)| . 

We need a set of generators for this subgroup. For any integer n > 1, define 
M{n) ^M{Z)nTg{n), U{n) ^ IJ{Z) nVgin), V{n) =V{Z) nTg{n). 

with 

Tgin) = {M eVg \ M= l2g(modn)} . 
Proposition A. 1.1. The subgroups M{1), U{2) and V{1) generate ^g{'2), and 

rO(2) = r<,(2).M(z).v(z). 



r^(2) = { 
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Proof. First, the subgroups M(2), U{2) and V{2) generate rg(2), see p. 179]. 
Let 



ri(2) = 



A B 
C D 



e Sp (Z) \ A = D= l(mod2) and B = 0(niod2) \ . 



There is the foUowing diagram, where the vertical arrow is the transpose of the 
reduction modulo 2: 

T,{2) c ri(2) C rO(2) C T,{1) 
C U(F2) C P(F2) C Spg(F2) 



Then, if M e T\{2) is written as usual 



'A 


B 




Ig 0' 




'A~ 


VBC 


B 


C 


D 




c V 






VDC 


D 



and if M e T° {2), then 



'A 


B 




'A-^ 0' 




Ig B'A 


C 


D 




^A 




CA-^ D'A 



er,(2), 



e ri(2). 



□ 



Theorem A. 1.2. Assume <? > 3. The function Xk{^T) is a modular form on 
of weight k for T^{2). 

Proof. J. -I. Igusa proved p. 850] that Xk{T) is a modular form of weight k for 
r3(l) if .9 > 3. Let 



H = 



1 



H.r = ir. 



Let /(r) = Xki^T) = Xk{H.T). It is sufficient to check that 

f{M.T)^j{M,Tff{T) 

if M belongs to one of the generating subgroups described in Prop. lA.l.ll First, if 
M G M(l), then H.M = M.H, hence, 

/(M.r) = XkiH.M.r) = XkiM.H.r) = j{M,H.Tfxk{H.T) = j{M,Tf fir), 

since j{M,T) = ±1 for every M G M(Z) does not depend on t G Hg. Now, if 
U G C/(2), then 



u ^u- 



12 



I3 2B 
I3 



where U' 



I3 B 
I3 



G U(Z), 



and H.U = H.U'^ = U'.H. This implies 

f{U.T)^Xk{H.U'\T)^Xk{U'.H.T)^j{U',H.Tfxk{H.T)^j{U,Tff{T), 

since j{U'\T) = 1 for every U G U(Z). If ^ G 1^(1), then H.V = V^.H. Hence 
fiV.T) = XkiH.V.T) = Xk{V\H.T) = ]{V\H.Tf Xk{H.T) 

^]{V,Tfxk{H.T)^j{y,Tff{T), 

since j(F^T) = 3{V,2t) for every V G 



□ 
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